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ABSTRACT 

This study reviewed the statistical practices in published 
research articles in the "Journal of Education for Students Placed at Risk" to 
determine the reporting of effect sizes and structure coefficients. Of the 12 
quantitative studies found in the last 3 volumes of the journal, only 3 were 
identified as using multiple regression analysis. Two of the three indicated 
use of effect size magnitudes, but the third reported only "F" and "p-" 
values. None of these articles gave structure coefficients. Reporting effect 
sizes highlights the difference between statistical and practical 
significance. The paper also recommends the use of "what-if" analysis when 
using regression. (SLD) 
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Multiple Regression Methodology in the Journal of Education for Students Placed at 
Risk. Effect Sizes and Structure Coefficients 

In any research study, the integrity of the research is only as strong as the weakest 
link. Study validity can be threatened from myriad fronts, including attacks to what is 
commonly called statistical conclusion validity. Statistical conclusion validity refers to 
the appropriate use and interpretation of the statistical methodology used in a given study 
(Babbie, 1992). When methods are not appropriately employed, the conclusions that can 
be drawn from a line of inquiry are weakened. Accordingly, the purpose of the present 
paper is to review the statistical practices in published research articles in the Journal of 
Education for Students Placed at Risk concerning two important issues in regression 
research. 

First, as in all general linear model analyses (cf. Cohen, 1968; Knapp, 1978) and 
consistent with current trends in best statistical practice (Thompson, 1999; Wilkinson & 
American Psychological Association (APA ) Task Force on Statistical Inference, 1999), 
the interpretation of effect sizes in regression research is critical to understanding one’s 
study results. Effect sizes help researchers understand the magnitude of relationships 
among their variables, and therefore shed light on result importance (Thompson, 1999). 
Effect size reporting is becoming increasingly important as researchers become aware of 
the limits of statistical significance testing. As the APA Task Force recommended: 

It is hard to imagine a situation in which a dichotomous accept-reject decision is 
better than reporting an actual />-value or, better still, a confidence 




3 



Effect Sizes and Structure Coefficients 



3 



interval. Always provide some effect-size estimate when reporting a/?-value (p. 

599, emphasis added). 

Second, and also in all general linear model analyses, structure coefficients are 
critical to interpreting variable importance. Indeed, the common practice of examining 
only standardized weights (i.e., beta weights) in regression research can lead to erroneous 
conclusions (Thompson & Borrello, 1985;Courville & Thompson, 2001). Because they 
represent the simple bivariate correlation between a predictor variable and the latent 
predicted variable (i.e. Yhat), structure coefficients tell us the contribution each predictor 
could have made in accounting for overall effect. Structure coefficients are not adversely 
affected by multicollinearity (as beta weights are), and therefore overcome the so-called 
“bouncing beta” problem in which the beta weights can dramatically change as predictor 
intercorrelations fluctuate across studies. Accordingly, both beta weights and structure 
coefficients should almost always be interpreted in regression research. 

Several studies have examined the reporting of effects and structure coefficients 
in research (cf. Courville & Thompson, 2001; Henson & Smith, 2000; Vacha-Haase, 
Nilsson, Reetz, Lance, & Thompson, 2000). In large part, and unfortunately, both effect 
sizes and structure coefficients are not frequently reported and interpreted in the research 
literature. Kirk (1996) suggested increase frequency of reporting for multiple regression 
articles due to the fact that statistical software often readily provide R 2 effects in 
regression output. 

Purpose 

To further examine the current status of best reporting practice, the present paper 
will examine the use and interpretation of structure coefficients and effect sizes in 
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multiple regression articles in the Journal of Education for Students Placed at Risk. 
Therefore, the present paper will extend the methodological examination to the 
multicultural and at-risk education literature. 

Articles using multiple regression were first identified in the last three volumes of 
the journal. These articles were then examined and coded concerning whether the article 
(a) reported structure coefficients, (b) interpreted structure coefficients, (c) reported effect 
sizes, and (d) interpreted effect sizes. Along with this data, the paper also documents the 
types of effect sizes used. Examples of best and poor practice are noted and 
recommendations for improved reporting practice are offered. 

Method 

Article Selection 

A total of twelve quantitative type studies were found in the last three volumes of 
The Journal of Education for Students Placed At Risk. Only three of those twelve articles 
were identified as using multiple regression analysis, of which this paper addresses. 

Effect Size Reporting 

Two of the three articles examined using multiple regression in The Journal of 
Education for Students Placed At Risk indicated use of effect size magnitudes. The third 
article only reported F and /7-values. 

Effect Size Interpretation 

Kirk (1996) identified 16 varieties of magnitude effect reporting in social science 
journals reviewed in one study, however only r, adjusted r 2 and eta 2 were interpreted in 
the articles reviewed that used multiple regression in The Journal of Education for 
Students Placed At Risk. Of the two studies using effect sizes, adjusted r 2 was only 
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referenced once; eta 2 was also reported in one article. The article reporting adjusted r , 
provided an interpretation of the data results in both the Results and Discussion sections, 
the other article reporting eta 2 provided an interpretation in the Results section only. 
Structure Coefficient Reporting and Interpretation 

In examining studies that used multiple regression analysis in The Journal of 
Education for Students Placed At Risk, all of those studies were quantitative in nature, 
reporting statistics such as the mean, standard deviation, F, and /7-values. However, none 
reported or interpreted structure coefficients or beta weights. 

Reporting and Interpreting Structure Coefficients 
Reporting structure coefficients as an important measure in research studies has 
yet to be found as a common method in examining and analyzing data, in the general 
linear model (Courville & Thompson, 2001). However, as noted by Thompson (1992), 
the reporting of beta weights and structure coefficients are both important. Thompson 
(1992) further argued that “the thoughtful researcher should always interpret either (a) 
both the beta weights and the structure coefficients or (b) both the beta weights and the 
bivariate correlations of the predictors with Y“ (p. 14). 

Unfortunately, in regression research, heavy emphasis has been placed on solely 
interpreting beta weights to the exclusion of structure coefficients. For example, one 
study (Courville & Thompson, 2001) noted that only three of twenty articles reviewed in 
The Journal of Counseling Psychology between January 1990 and April 1993 reported 
structure coefficients. 

Since structure coefficients can help explain the predictive importance of a 
predictor variable’s contribution to the overall effect, reporting these measures may help 
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in explaining the theoretical and practical ramifications of the results. Courville and 
Thompson (2001) suggested that if a general linear model analysis is used in a study 
where predictors are correlated, as they often are, then “when interpreting regression 
results, once noteworthy effects have been detected it may be best to consult the full 
system of results, just as we routinely would in applications of other members of the 
general linear model analytic family” (p. 45). 

A structure coefficient is a correlation between an observed variable (e.g. 
predictor) and a synthetic variable (e.g. Yhat); an example structure coefficient formula 
for predictor XI would be: r structure = r Y with Xl/R (cf. Courville & Thompson, 

2001; Daniel & Onwuegbuzie, 2001). Structure coefficients are not readily obtainable 
for multiple regression using a statistical software package (i.e. SPSS), although it is as 
much a correlation analysis as Pearson’s r, and can be calculated using the statistical 
package (cf. Daniel & Onwuegbuzie, 2001). All researchers need to do is to save the 
unstandardized predicted values (i.e. Yhat) as part of the regression analysis and then 
correlate these Yhat scores with the observed predictors. When one squares the structure 
coefficient (r s 2 ), the result is the percentage of variance the predictor could contribute to 
the explained effect. 

Reporting Effect Sizes 

Reporting effect sizes alongside /7-values, given the limitations of statistical 
significance testing, is long overdue for educational journals (Kirk, 1996). As noted by 
Kirk (1996), “researchers have been encouraged to supplement reports of null hypothesis 
tests with measures of effect magnitude” for over 70 years now. 




7 



Effect Sizes and Structure Coefficients 



7 



Effect sizes can generally be grouped into two categories. Category one 
includes mean difference effects such as Cohen’s d, which represents the difference 
between experimental and control groups divided by the pooled standard deviation of 
both groups. Category two includes variance-accounted-for effects such as r 2 , eta 2 and 
adjusted r 2 (Snyder & Lawson, 1993). 

Reporting an effect size is necessary in that statistical significance testing alone 
only answers the question “If the sample represents the population, how likely is the 
obtained [sample] result?” (Thompson, 1998, p. 34, emphasis in original). Too, statistical 
conclusion on the hypothesis is a function of obtained p-values, which, as Henson and 
Smith suggested (2000, p.286, emphasis in original) only “indicate the probability (0 to 
1 .0) of attaining the presently observed results from the present sample assuming that the 
null hypothesis is exactly true in the population.” This makes it necessary to supplement 
such judgments with alternative reporting methods, such as effect size measures to 
indicate the magnitude of the finding, not just the likelihood of the finding. 

According to Kirk (1996), reporting a statistic that indicates a specific magnitude 
of effect, rather than reporting a general reference to the use of effect sizes, is more 
favorable. He further noted that r 2 and eta 2 are readily recorded as such, when using 
statistical software packages, such as SPSS (Kirk, 1996). 

Practices and Recommendations 

Of the articles reviewed in The Journal of Education for Students Placed At Risk 
using multiple regression, three of them reported p-values or F statistics. Of these 
however, only two offered effect sizes magnitude; none gave structure coefficients. 
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It is poor practice to report statistical significance testing values without also 
reporting and interpreting effect size values (Thompson, 1998). Thompson also posited 
“most authors will simply not change their practices until editorial policies require them 
to do so (Thompson, 1998, p. 35). In this same article, he also confronted the wording of 
the American Psychological Association’s 4 th edition publication manual in reference to 
reporting practices for effect sizes of “encouraging” authors to report effect sizes; he 
cited this wording as “too vague to enforce “ (Thompson, 1998, p. 35). It is interesting to 
note here that the APA’s 5 th edition (2001) now reads: 

For the reader to fully understand the importance of your findings, it is almost 
always necessary to include some index of effect size or strength of relationship 
in your Results section. You can estimate the magnitude of effect or the strength 
of the relationship with a number of common effect size estimates, including (but 
not limited to) r 2 , rj 2 , to 2 , R 2 , tp 2 , Cramer’s V, Kendall’s W, Cohen’s d and k, 
Goodman-Kruskal’s k and y, Jacobson and Truax’s (1991) and Kendall’s (1999) 
proposed measure of clinical significance, and the multivariate Roy’s 0 and the 
Pillai-Barlett V (pp. 25-26, emphasis added). 

The Journal of Education for Students Placed At Risk shows evidence in the quantitative 
articles reviewed that effect size reporting and interpretation is practiced. It must be 
noted her that The Journal of Education for Students Placed At Risk does include both 
quantitative and qualitative articles. Of these articles (forty-five), however, only twelve 
of them were quantitative studies. In examining the twelve quantitative studies, only 
three used multiple regression, but of these three, two of them reported effect sizes. 
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The practice of reporting structure coefficients in The Journal of Education for 
Students Placed At Risk, is not popular to say the least (none were noted). Courville and 
Thompson (2001) cited “in most cases, regression researchers ought to interpret P 
weights and structure coefficients (or else bivariate correlations of predictors with the 
criterion) once a noteworthy omnibus effect is detected” (p. 244). Also, Courville and 
Thompson (2001) noted the importance of interpreting structure coefficients in that the 
reason for calling “structure coefficients” such is “that they provide insight regarding 
what is the nature or structure of the underlying synthetic variables of the actual research 
focus, (p 238, emphasis added). 

Recommendat ions 

It is recommended that authors report /^-values and test statistics from the general 
linear model when necessary, but also include the reporting and interpreting of effect 
sizes to explain magnitude of effect. When notable effects are observed, then and only 
then, report and interpret both standardized weights and structure coefficients to explain 
how much each predictor is contributing to the model (Daniel & Onwuegbuzie, 2001). 
Reporting effect sizes also “highlight the distinction between statistical and practical 
significance. Results that are statistically significant are not always noteworthy in a 
practical sense” (Snyder & Lawson, 1993, pp. 334-335, emphasis in original) but effect 
sizes “tells us how much of the dependent variable can be controlled, predicted, or 
explained by the independent variable(s)” (Snyder & Lawson, 1993, p. 335). 

It is also recommended that researchers conduct a newer, precise “what-if 
analysis” when using regression, as noted by Henson and Smith (2000) and proposed by 
Kieffer and Thompson ( 1 999); this newer “what-if analysis,” provides for corrected 
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effect estimates (Henson & Smith, 2000). A “what if’ analysis resembles a post hoc 
power analysis and helps researchers place statistical significance tests in the context of 
sample size (Henson & Smith, 2000). 

Because most researchers learn their methodology skills in educational statistics 
courses in university settings, it is recommended that courses and textbooks reflect such 
practices. Thompson (1998, emphasis in original) noted “ empirical studies consistently 
show that many researchers do not fully understand the logic of statistical tests” (p. 34). 
He further stated, “Similarly, many textbooks teach misconceptions regarding these tests 
(Thompson, 1998, p. 34). 

Critical to statistical conclusion validity when reporting /7-values is to always 
report and interpret effect sizes. To analyze how much a variable(s) is contributing to 
that effect, researchers should almost always report structure coefficients. Conducting a 
“what-if ’ analysis when interpreting data results and reporting statistical significance 
findings in terms of “sample size,” also helps facilitate understanding of results. As 
suggested by Thompson (1997): 

All analyses are part of one general linear model. When interpreting results in the 
context of this model, researchers should generally approach the analysis 
hierarchically, by asking two questions: 

- Do I have anything? (Researchers decide this question by looking at some 
combination of statistical significance tests, effect sizes... and replicability 
evidence.) 
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If I have something, where do my effects originate? (Researchers often 
consult both the standardized weights implicit in all analyses and 
structure coefficients to decide this question.) (p.31) 
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